AL/HR-TP-1 997-0003 


UNITED  STATES  AIR  FORCE 
ARMSTRONG  LABORATORY 


AN  ASSESSMENT  METHODOLOGY 
FOR  TEAM  COORDINATION 
IN  COMBAT  MISSION  TRAINING 


Steven  J.  Tourville 

Hughes  Training,  inc.,  Training  Operations 
6001  South  Power  Road,  Buiiding  560 
MesaAZ  85206-0904 

V.  Alan  Spiker 


Anacapa  Sciences,  inc. 

301  E.  Carriiio  Street 
Santa  Barbara  CA  93101 

Denise  R.  Silverman 

Hughes  Training,  Inc.,  Training  Operations 
6001  South  Power  Road,  Building  560 
MesaAZ  85206-0904 


Robert  T.  Nullmeyer 


ARMSTRONG  LABORATORY 
HUMAN  RESOURCES  DIRECTORATE 
AIRCREW  TRAINING  RESEARCH  DIVISION 
6001  South  Power  Road,  Building  561 
Mesa  AZ  85206-0904 


July  1997 


I 


Approved  for  public  release;  distribution  is  unlimited. 


AIR  FORCE  MATERIEL  COMMAND 
ARMSTRONG  LABORATORY 
HUlWAN  RESOURCES  DIRECTORATE 
7909  Lindbergh  Drive 
Brooks  Air  Force  Base,  TX  78235-5352 


19970808  024 


NOTICES 


When  Government  drawings,  specifications,  or  other  data  are  used  for 
any  purpose  other  than  in  connection  with  a  definitely  Government-related 
procurement,  the  United  States  Government  incurs  no  responsibility  or  any 
obligation  whatsoever.  The  fact  that  the  Government  may  have  formulated  or 
in  any  way  supplied  the  said  drawings,  specifications,  or  other  data,  is  not  to 
be  regarded  by  implication,  or  otherwise  in  any  manner  construed,  as  licensing 
the  holder,  or  any  other  person  or  corporation;  or  as  conveying  any  rights  or 
permission  to  manufacture,  use,  or  sell  any  patented  invention  that  may  in  any 
way  be  related  thereto. 

The  Office  of  Public  Affairs  has  reviewed  this  paper,  and  it  is  releasable 
to  the  National  Technical  information  Service,  where  it  will  be  available  to  the 
general  public,  including  foreign  nationals. 

This  paper  has  been  reviewed  and  is  approved  for  publication. 


ROBERT  T.  NULLMEYER  DEE  H.  ANDREWS 

Project  Scientist  Technical  Director 


LYNN  A.  CARROLL,  Colonel,  USAF 
Chief,  Aircrew  Training  Research  Division 


Please  notify  AL/HRPP,  7909  Lindbergh  Drive,  Brooks  AFB,  TX  78235-5352,  if  your 
address  changes,  or  if  you  no  longer  want  to  receive  our  technical  reports.  You  may 
write  or  call  the  STINFO  Office  at  DSN  240-3877  or  commercial  (210)  536-3877. 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

0MB  No.  0704-0188 

anH  infomiation  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources  oatherina 

and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  irrformation.  Send  comments  regarding  this  burden  estimate  or  anv  other  asoect  of  this  collection  of  ^ 

l”204^Ari?nntQn  f Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  jSferson  Davis  Highway  Suite 

1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget.  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503.  * 

1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

July  1997  Final  -  Sentember  1995  to  nerembfir  1 OQA 

4.  TITLE  AND  SUBTITLE 

An  Assessment  Methodology  for  Team  Coordination  in  Combat  Mission  Training 

5.  FUNDING  NUMBERS 

C  -  F41624-95-C-5011 

PE  -  62202F 

PR  -  1123 

TA  -  B2,  B3 

WU-  06,  01 

6.  AUTHOR(S) 

Denise  R.  Silverman  Steven  J.  Tourville 

V.  Alan  Spiker  Robert  T.  Nullmeyer 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Hughes  Training,  Inc.  Anacapa  Sciences,  Inc. 

Traming  Operations  30 1  E.  Carrillo  Street 

600 1  South  Power  Road,  Building  56 1  Santa  Barbara  CA  93 1 0 1 

Mesa,AZ  85206-0904 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Armstrong  Laboratory 

Human  Resources  Directorate 

Aircrew  Training  Research  Division 

6001  South  Power  Road,  Building  558 

Mesa,AZ  85206-0904 

10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

AL/HR-TP- 1997-0003 

11.  SUPPLEMENTARY  NOTES 

Armstrong  Laboratory  Technical  Monitor:  Dr  Robert  T.  Nullmeyer,  (602)  988-6561 

This  technical  paper  has  been  published  in  the  Proceedings  of  the  18th  Interservice/Industiy  Training  Systems  and  Education 

Conference,  held  2-4  December  1996  in  Orlando  FL.OrlandoFL 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

A  Team  Mission  Obsepation  Tool  (T-MOT)  was  developed  to  identify  individual  and  team  behavioral  processes  observed 
durmg  a  specialized,  sunulation-based  program  of  Combat  Mission  Traming  (CMT)  conducted  for  U.S.  Air  Force  Snecial 
Operations  Command  (SOC),  MD-130P  Special  Operations  Forces  (SOF)  aircrew  teams.  The  T-MOT,  its  foundatioL 
developinent  Md  purpose  are  described.  Measurement  is  accomplished  within  the  T-MOT  using  behaviorally  anchored  ratine 
scales  and  subject  matter  expert  observations  of  key  behaviors  tied  to  a  complex  CMT  scenario.  The  T-MOT  supports  ^ 

recordmg  and  ^alysis  of  both  mdividual  and  aircrew  team  behaviors  within  five  Crew  Resource  Management  (CRM) 
subprocesses  (tune  management;  tactics  employment;  function  allocation;  situation  awareness;  and  command  control  and 
communicabons)  across  critical  mission  phases.  Additionally,  the  T-MOT  provides  structure  to  direct  observations  of 
complex  perfomances  demonstrated  during  both  mission  preparation  and  mission  execution.  With  this  methodology  an 
mtemally  consistent  and  reliable  "record  by  exception"  measurement  philosophy  for  recordmg  specific  aircrew  team  mission 
behaviors  demonsfrated  during  CMT  is  provided.  The  T-MOT  is  being  used  to  address  several  research  questions:  (a)  Are 
tern  behaviors  within  one  or  more  CRM  subprocess  areas  related  to  overall  mission  performance'?  (b)  Which  CRM 
subprocess  areas  have  the  greatest  demonstrated  impact  on  mission  outcome?  (c)  Is  team  performance  related  to  mission 
outcome  above  Ae  perfoimance  represented  by  each  crew  position?  and  (d)  Do  effective  aircrew  teams  exhibit  consistent  sets 
of  coordination  behaviors  that  can  be  "captured"  and  designed  into  a  CMT  program?  With  this  assessment  approach  team 
coordmation  process  indices  have  been  identified  for  emphasis  using  current  CMT  technologies;  a  scheme  for  improvinc^  team 

existmg  capabilities  was  identified;  and  CMT  system  effectiveness  was  assessed  Additionally 
me  T-MOT  h^  demonstrated  the  potential  to  be  expanded  to  other  CMT  environments  with  only  modest  modification  and  can 
be  viewed  as  the  first  step  in  the  development  of  an  overall  team  mission  readiness  assessment  tool. 

14.  SUBJECT  TERMS 

Aircrews;  CMT;  Combat  mission;  Combat  mission  Training;  Crew  resource  management; 
CRM;  Performance;  Readiness  assessment;  Team  behaviors;  Team  coordination;  Team  ’ 
performance;  Teams;  Training; 

15.  NUMBER  OF  PAGES 

16 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION 
OF  REPORT  OF  THIS  PAGE  OF  ABSTRACT 

Unclassified  Unclassified  Unclassified 

20.  LIMITATION  ABSTRACT 

UL 

NSN  7540-01-280-5500  Standard  Form  298  (Rev  2-89) 

Prescribed  by  ANSI  Std  Z-39-18 


CONTENTS 


PAGE 

INTRODUCTION .  1 

THE  T-MOT  AND  ITS  RELEVANT  DOMAINS .  1 

Mission  Phases  .  2 

CRM  Subprocesses  . ; .  2 

T-MOT  Content  Identification  .  3 

THE  T-MOT  APPROACH  .  3 

Key  Behaviors  .  3 

Multi-Measure,  Multi-Method  Approach  .  4 

Observation  Procedures  .  4 

Rating  Procedures  .  4 

THE  T-MOT  METHOD 

T-MOT  APPLICATION  .  g 

Research  Questions  .  g 

Methodological  Constraints  .  9 

IMPLICATIONS .  10 

REFERENCES  .  1 1 

FIGURES 

Figure 

No. 

1  SME  Ratings  Matrix  for  the  Mission  Preparation  (MP)  Phase .  5 

2  Mission  Preparation  (MP)  Phase  (Example)  .  5 

3  Low-Level  (LL)  Tactical  Operations  (Example)  .  5 

4  Aerial  Refueling  (AR)  Operations  (Example)  .  6 

5  AirDrop  (AD)  Operations  (Example)  .  5 

6  Infil/Exfil  (IE)  Operations  (Example)  .  6 

7  Situation  Awareness  (SA)  Functional  CRM  Subprocess  Area  (Example)  .  6 

8  Fimction  Allocation  (FA)  Functional  CRM  Subprocess  Area  (Example)  .  7 

9  Tactics  Employment  (TE)  Functional  CRM  Subprocess  Area  (Example)  .  7 

10  Time  Management  (TM)  Fimctional  CRM  Subprocess  Area  (Example)  .  7 

1 1  Command,  Control,  and  Communications  (C3)  Functional 

CRM  Subprocess  Area  (Example)  .  7 

hi 


PREFACE 


This  report  documents  a  paper  which  was  presented  at  the  Interservice/Industry 
Training  Systems  and  Education  Conference  (l/ITSEC)  which  was  held  in  Orlando  FL 
from  2-4  December  1996.  This  research  was  conducted  underwork  Unit  1123-B2-06, 
Aircrew  Training  Research  Support;  Contract  F41624-95-C-5011  with  Hughes  Training, 
Inc-Training  Operations;  and  1123-B3-01,  Special  Operations  Forces  (SOF)  Aircrew 
Training  and  Mission  Preparation  Research.  The  Laboratory  Principal  Investigator  was 
Dr  Robert  T.  Nullmeyer;  the  Laboratory  Contract  Monitor  was  Mr  Daniel  Mudd. 

This  effort  is  part  of  an  Armstrong  Laboratory,  Human  Resources  Directorate, 
Aircrew  Training  Research  Division  (AL/HRA)  program  to  help  identify  how  mission 
rehearsal  devices  can  be  most  effectively  used  in  mission  rehearsal,  and  to  assess  the 
actual  impact  of  this  technology  as  it  becomes  available  in  the  Air  Force  SOF 
community. 


AN  ASSESSMENT  METHODOLOGY  FOR 
TEAM  COORDINATION  IN  COMBAT  MISSION  TRAINING 


INTRODUCTION 

A  great  percentage  of  fatal  aircraft  incidents  and  accidents  are  attributable  to  human  error  caused  by  poor  aircrew 
coordination.  The  perceived  need  to  reduce  human  error  in  Crew  Resource  Management  (CRM)  training  led  to  the 
development  of  a  unique,  mission-specific  Combat  Mission  Training  (CMT)  program  for  USAF  MC-130P  (formerly 
HC-130P/N)  Special  Operations  Forces  (SOF)  aircrews.  This  training  is  currently  managed  by  the  USAPs  58th 
Training  Support  Squadron  (TRSS),  and  conducted  by  the  Mission  Training  Support  System  (MTSS)  at  Kirtland  AFB, 
NM. 

CRM  training  serves  as  the  foundation  of  CMT  by:  (a)  Identifying  to  SOF  aircrews  those  behavioral  skills  that  are 
critical  to  aircrew  coordination  in  complex  mission  operations,  and  (b)  developing  strategies  and  procedures  for 
training  these  skills.  While  the  need  for  such  training  is  widely  accepted,  data  regarding  an  aircrew's  coordination 
effectiveness  in  CMT  is  largely  unavailable. 

Both  CRM  and  CMT  are  high-priority,  annual  training  events  for  SOF  aircrews.  While  some  operational  flying  tasks 
are  performed  by  only  one  individual,  the  vast  majority  of  mission  operations  are  performed  by  collective  aircrew 
teams.  A  wide  diversity  of  roles  and  responsibilities  are  present  in  a  typical  SOF  MC-130P  mission.  The  MC-130P 
mission  crew  consists  of  the  aircraft  commander  (AC),  co-pilot  (CP),  flight  engineer  (FE),  left  and  right  navigators 
(LN,  RN),  and  communications  system  operator  (CSO).  In  addition,  there  are  other  members  of  the  combat  mission 
team  whose  tactical  actions  directly  affect  the  aircrew,  such  as  any  Special  Forces  “customers,”  or  other  Air  Force 
Special  Operations  (AFSOC)  aircraft  that  will  be  participating  in  the  mission  (e.g.,  a  helicopter  that  is  to  be  refueled 
by  an  MC-130P).  Besides  the  supporting  and  tactical  players,  the  team  also  includes  those  individuals  and  functions 
who  provide  a  command  and  control  role,  such  as  the  command  leadership  and  the  airborne  command,  control,  and 
communication  (ABCCC)  aircraft.  These  teams  must  be  trained  to  coordinate  their  specialized  activities  in  order  to 
produce  effective  decision-making  and  successful  mission  performance 

The  evaluability  of  CRM  coordination  subprocesses  is  increased  when  these  have  been  defined  in  terms  of  tasks  to 
be  trained.  A  well-planned  and  conducted  front-end  analysis  that  establishes  evaluation  criteria  contributes  to  the 
program's  overall  evaluability.  CRM  and  CMT  training  programs  have  typically  been  designed  without  regard  to  such 
evaluation  concerns  due  to  the  difficulty  in  establishing  sensitive  indices  of  crew  mission  performance  (Silverman, 
Spiker,  Tourville,  &  Nullmeyer,  1996). 

A  major  goal  of  our  research  program  is  the  identification,  measurement,  validation,  and  eventual  reinforcement  of 
aircrew  behaviors  associated  with  combat  mission  readiness  and  effective  mission  performance.  The  specific 
purpose  of  this  paper  is  to  report  on  one  component  of  our  total  assessment  strategy.  A  Team-Mission  Observation 
Tool  (T-MOT)  has  been  developed  to  determine  an  MC-130P  aircrew  team’s  coordination  effectiveness.  Two  issues 
were  applicable  to  the  development  of  this  methodology:  (1)  What  are  the  relevant  processes  associated  with 
effective  team'  coordination  in  the  SOF  CMT  program?  and,  (2)  What  specific  evaluation  components  and 
associated  behavioral  criteria  must  be  included  in  the  T-MOT? 

THE  T-MOT  AND  ITS  RELEVANT  DOMAINS 

To  obtain  background  data  on  CRM,  team  processes,  CMT  concepts,  and  related  research  programs,  relevant 
documents  were  acquired  from  a  broad  base  of  available  sources,  and  were  reviewed  during  the  initial  phases  of  the 
project.  In  particular,  documents  were  collected  relating  to:  (a)  SOF  concept  of  mission  operations,  (b)  SOF  service 
unit  doctrinal  statements,  (c)  specific  SOF  employment  procedures,  (d)  SOF  mission  tactics,  and  (e)  SOF  training 
task  and  objectives  documents.  The  relevant  data  were  content-analyzed  to  provide  a  solid  foundation  for 
understanding  SOF  CMT  and  CRM  issues. 

Given  our  emphasis  on  CRM  and  team  performance  measurement,  and  our  interest  in  the  MC-130P  weapon 
system,  two  regulatory  requirements  were  of  particular  importance:  (a)  the  mission  definition  portion  of  AFSOC  Reg. 
51-130  (1994),  and  (b)  the  Air  Force  Instruction  for  CRM  Training  -  AFl  36-2243  (USAF,  1994).  The  first  document 
outlines  specific  CMT  tasks  to  be  performed,  and  the  second  document  provides  guidance  on  methods  that  can  be 
used  to  train  and  measure  CRM  task  performance. 
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The  AFSOCR  51-130  (Flying  Training  Regulation),  served  as  a  primary  source  by  establishing  which  mission  events 
are  to  be  trained  and  whether  these  are  being  accomplished  to  a  satisfactory  extent.  By  identifying  critical  mission 
training  events  and  success  factors,  this  document  formed  the  basis  for,  and  established  the  validity  of,  the  mission 
scenario  that  is  used  to  conduct  CMT  in  the  MC-130P  Weapon  System  Trainer  (WST). 

The  AFl  36-2243  emerged  as  the  second  important  document,  establishing  a  requirement  for  Major  Commands  to 
measure  CRM.  The  AFl  specifies  definite,  albeit  broad,  boundaries  around  the  content  areas  within  which  our  own 
data  collection  efforts  are  formulated.  The  AFl  Is  an  essential  element  of  CRM  training,  as  it  highlights,  for  example, 
the  importance  of  situation  awareness  (SA)  and  mission  planning  in  CRM,  expands  the  scope  of  CRM  concepts  to 
the  tactical  environment,  and  permits  over-the-shoulder  observation  as  a  legitimate  data  collection  technique. 

Finally,  information  from  personnel  who  were  experienced  in  SOF  operations,  mission  execution,  and  training  also 
served  as  data  sources.  Data  were  collected  using  informal  interviews,  observations  of  training  events,  and  group 
discussion  techniques.  Besides  providing  information  regarding  SOF  mission  execution  and  training  assessment 
issues,  subject  matter  expert  (SME)  interviews  provided  clarification  on  issues  not  addressed  in  the  doctrinal 
publications. 

Mission  Phases 

The  mission  events  identified  by  AFSOCR  51-130  formed  the  basis  for  categorizing  five  mission  “phases,”  in  which 
there  are  multiple  situations  where  crewmembers  would  be  expected  to  engage  in  specific  CRM  behaviors.  These 
mission  phases,  and  our  description  of  their  CMT  objectives,  follows; 

The  Mission  Preparation  (MP)  phase  entails  conducting  pre-mission  planning  and  briefing  activities  that  allow 
sufficient  preparation  of  a  comprehensive  pre-mission  execution  plan.  This  plan  will  be  prepared  with  considerations 
for  a  medium  threat  environment,  all  major  mission  events  and  activities,  and  mission  operations  procedural 
constraints. 

The  Low-Level  (LL)  tactical  operations  phase  includes  night  vision  device  (NVD)  low-level  flight  en  route  to 
specific  mission  events,  using  proper  tactical  mission  management  procedures  (altitude,  airspeed,  terrain  masking, 
etc.)  for  a  medium-threat  environment. 

An  Aerial  Refueling  (AR)  operations  phase  involves  the  successful  conduct  of  tactical  in-flight  AR  of  (multiple) 
MH-53J  Pave  Low  helicopters  within  prescribed  time,  course,  and  altitude  constraints  in  a  medium-threat 
environment. 

The  AirDrop  (AD)  operations  phase  involves  conducting  a  personnel  Computed-Air-Release-Point  (CARP) 
airdrop  within  prescribed  time,  course,  and  altitude  constraints  in  a  medium-threat  environment. 

The  Infil/Exfil  (I/E)  operations  phase  includes  covert  infiltration  and/or  exfiltration,  at  tactical  landing  sites  for 
transload  purposes,  within  prescribed  time,  course,  and  altitude  constraints  in  a  medium-threat  environment. 

CRM  Subprocesses 

We  have  chosen  to  define  functional  CRM  subprocess  areas  based  on  considerations  of  specific  aircrew  tasks.  This 
is  in  contrast  to  defining  functional  areas  based  on  either  global  dimensions  of  performance  (e.g,,  CRM  meta-skills) 
or  situation-specific  considerations  (e.g.,  critical  behaviors).  In  particular,  since  our  ultimate  goal  is  to  elucidate  the 
coordination  processes  that  contribute  to  effective  aircrews  and  good  team  performance,  we  selected  functional 
areas  based  on:  (a)  relevance  to  the  SOF  mission  environment  and  previously  reported  operational  problems,  (b) 
appropriateness  to  the  high  levels  of  experience  and  motivation  of  most  MC-130P  aircrews,  (c)  applicability  to  CMT, 
and  (d)  amenability  to  measurement  by  outside  observers.  In  addition,  and  where  possible,  we  attempted  to  derive 
functional  areas  that  tap  the  more  global  CRM  dimensions  identified  by  other  researchers. 

Based  on  these  considerations,  five  functional  coordination  subprocess  areas  were  identified.  Our  description  of 
each  CRM  coordination  subprocess  and  rationale  for  their  selection  follows.  These  areas  are  to  be  assessed  during 
each  of  the  five  mission  phases  previously  discussed. 

In  reviewing  the  scope  and  content  of  these  areas,  it  is  evident  that  some  of  the  traditional  dimensions  of  CRM,  such 
as  leadership,  group  cohesiveness,  personalities,  etc.,  have  been  omitted.  We  readily  acknowledge  that  these  five 
areas  by  no  means  encompass  the  entire  content  domain  of  what  would  properly  be  considered  team  coordination, 
and  that  other  factors  are  worthy  of  study  in  their  own  right.  However,  our  focus  is  on  identifying  CRM  functional 
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areas  whose  performance,  we  think,  have  the  most  direct  links  to  the  tactical  CMT  environment,  training-related 
processes,  and  training  interventions: 

Situation  Awareness  (SA)  entails  maintaining  an  accurate  mental  picture  of  tactical  mission  events  and 
objectives  as  they  unfold  over  time  and  space.  Emphasis  and  analysis  are  placed  on  three  levels  of  SA  (perception, 
integration,  and  generation:  Endsley,  1995)  and  their  impact  on  team  coordination. 

Function  Allocation  (FA)  includes  the  division  of  crew  responsibilities  so  that  workload  is  distributed  among  the 
crew,  avoiding  redundant  tasking,  task  overload,  and  crewmember  disinterest  or  noninvolvement.  Tasks  should  be 
allocated  in  such  a  manner  so  that  crewmembers  are  able  to  share  Information  and  coordinate  responsibilities. 

Tactics  Employment  (TE)  includes  all  analytic  and  tactical  activities  necessary  to  avoid  or  minimize  threat 
detection  or  exposure,  and  to  successfully  coordinate  complex  mission  events  and  multiple  mission  objectives. 

Time  Management  (TM)  involves  the  ability  of  the  combat  mission  team  to  employ  and  manage  limited  time 
resources,  so  that  all  tasks  receive  sufficient  time  to  be  performed  correctly  and  critical  tasks  are  not  omitted. 

Command  Control,  and  Communications  (C3)  encompasses  activities  required  to  involve  external  parties  in 
the  mission;  maintain  communications  with  these  external  team  members;  monitor  internal  communications  within 
the  crew;  and  control  of  the  sequence  of  mission  events  according  to  the  mission  execution  plan. 

T-MOT  Content  Identification 

Preliminary  evaluation  criteria  and  the  practical  requirements  for  the  assessment  tool  were  established  after 
observing  a  series  of  SOF  CMT  classes.  The  purpose  was  to  collect  CMT  data  during  actual  instruction  through 
close-up  observation  of  training  and  SOF  mission  execution.  These  observations  allowed  direct  data  collection  from 
SOF  crewmembers  who  had,  during  training,  an  opportunity  to  articulate  their  reasons  behind  the  critical  issues 
related  to  operational  CMT  concerns. 

Content  analyses  were  then  performed  on  the  compiled  information  to  identify  the  major  components  and  topics 
presented  in  the  sets  of  data  arising  from  the  interviews  and  observations.  The  objective  was  to  identify  and  classify 
the  most  frequently  occurring  issues,  and  to  reconcile  analytic,  evaluative,  or  behavioral  discrepancies  and 
disagreements  in  information  provided  by  individual  experts. 

Finally,  the  draft  T-MOT  was  developed.  An  iterative  sequence  of  identification,  development,  and  refinement  was 
used,  wherein  the  logic  and  organization  of  the  T-MOT  were  scrutinized  for  consistency,  clarity,  and  coherence. 
Revisions  were  made  based  on  review  recommendations.  As  each  iteration  of  the  instrument  was  completed.  It  was 
compared  with  previous  versions  to  ensure  that  all  gaps  in  logic,  flow,  and  content  were  addressed.  SME  reviews 
were  used  throughout,  and  forma!  summative  reviews  were  conducted  at  each  development  milestone. 

THE  T-MOT  APPROACH 

The  T-MOT  approach  has  four  distinguishing  characteristics:  (a)  a  focused  perspective  on  key  behaviors  that  are 
collectible,  variable  across  aircrews,  and  operationally  relevant;  (b)  a  portable,  multi-method  and  multi-measure  mix 
of  variables  that  captures  complex  cognitive  processes;  (c)  a  naturalistic  observation-correlational  design  that  takes 
full  advantage  of  ongoing  training  systems,  resources,  and  a  readily  available  operational  subject  pool:  and  (d)  a 
unique  derivative  of  a  Behavioraliy  Anchored  Ratings  Scale  (BARS)  approach,  wherein  explicit  written  descriptions 
of  observed  behaviors  function  as  overt  referents,  and  aid  SME  scoring  decisions  when  determining  a  scale  value. 

Key  Behaviors 

Robust  measures  of  team  coordination  are  needed  that  must  capture  specific  key  behaviors  which  are  collectible, 
variable  across  crews,  and  operationally  relevant.  In  the  CMT  environment,  the  constraints  of  the  training  situation 
and  the  resources  available  must  be  considered.  The  T-MOT  measures  behaviors  that  we  reasonably  expect  to 
occur  on  a  regular  basis.  The  behaviors  selected  for  observation  must  also  be  variable  across  aircrews.  Given  the 
overarching  objective  of  identifying  effective  SOF  aircrews,  behaviors  that  maximally  differentiate  strong  from  weak 
crews  must  be  sought.  Preliminary  testing  and  SME  interviews  provided  insights  regarding  the  highest  payoff  areas 
(e.g.,  the  five  coordination  subprocesses)  and  potential  key  behaviors  on  which  to  focus. 

Next,  the  observed  behaviors  must  be  relevant  to  tactical  mission  operations.  Operational  realism  was  one  of  the 
primary  considerations  in  selecting  the  five  team  coordination  subprocesses  for  development  in  the  T-MOT. 
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Crewmembers  often  complain  about  the  “soft”  topics  traditionaliy  taught  in  CRM  courses  and  their  weak  connection 
to  the  missions  crews  actually  fly.  The  subprocesses  identified  in  the  T-MOT  attempt  to  bring  crew  coordination 
training  closer  to  the  operational  environment,  and  to  include  tactically  relevant,  behavioral  indices  of  team 
coordination.  These  behaviors  may  then  be  folded  back  into  training,  providing  crews  with  immediate  and  relevant 
feedback. 

Multi-Measure,  Multi-Method  Approach 

The  study  of  team  coordination  is,  by  its  very  nature,  a  multi-faceted,  multi-dimensional  problem.  Thus,  a  multi¬ 
measure,  multi-method  mix  of  variables  is  required  to  achieve  a  comprehensive,  systematic  investigation  of  the 
topic.  As  used  in  the  T-MOT,  this  approach  refers  to  the  employment  of  a  battery  of  objective  (e.g.,  counts)  and 
subjective  (e.g.,  ratings)  measures  coupled  with  quantitative  and  qualitative  methods  of  data  analysis. 

From  an  experimental  perspective,  this  approach  is  appealing,  as  it  permits  researchers  to  tap  cognitively  complex 
processes  that  may  otherwise  be  difficult  to  capture  within  a  single  index.  Logistically,  this  approach  has  a  robust 
appeal  with  regard  to  potentially  devastating  losses  of  partial  data  due  to  simulator  malfunctions  or  subject-crew 
turbulence.  In  the  T-MOT,  team  coordination  processes  are  observed  and  rated  by  an  SME  across  the  five 
coordination  subprocesses,  across  the  five  phases  of  flight,  and  across  the  six  crewrhembers.  This  is  accomplished 
by  using  headsets  to  monitor  live  mission  execution  and,  as  discussed  below,  from  over-the-shoulder  observations 
during  mission  preparation. 

Observation  Procedures 

An  over-the-shoulder  type,  unstructured  observation  technique  is  advocated  for  CRM  evaluation  in  the  AFI  36-2243. 
This  technique  has  been  successfully  employed  in  previous  mission  planning  and  mission  rehearsal  (Spiker  & 
Nullmeyer,  1995a,  1995b)  studies.  There  is  no  script  for  the  observer,  and  the  observer  is  free  to  record  overt 
behaviors.  Our  measurement  approach  is  to  record  "by  exception,"  where  the  observer  notes  both  individual 
crewmember  and  aircrew  team  behaviors  and  cognitions  that  seem  unusually  strong  or  weak,  as  compared  to  teams 
observed  in  the  baseline  period.  With  this  approach,  we  intend  to  capture  occurrences  of  effective  or  ineffective 
crew  coordination  behaviors  demonstrated  during  CMT.  Once  collected,  content  analyses  may  be  performed  on  the 
recorded  behaviors  to  permit  comparisons  of  frequency,  quality,  and/or  intensity  across  teams.  This  qualitative 
analysis  then  supplements  the  quantitative  analyses  performed  on  the  rating  data. 

For  our  initial  research  efforts,  we  have  employed  a  naturalistic  observation-correlational  design  rather  than 
performing  an  explicit  experimental  manipulation.  There  are  several  reasons  for  this  choice.  First,  this  approach 
allows  us  to  take  advantage  of  ongoing  CMT,  using  a  high-workload  combat  mission  scenario  that  is  already  in 
place.  By  working  within  the  CMT  community  on  a  not-to-interfere  basis,  we  have  gained  access  to  an  experienced, 
inexpensive  subject  pool.  Second,  use  of  a  naturalistic  observation  paradigm  offers  the  advantages  of  true 
operational  relevance  (external  validity)  and  clear-cut  application  of  team  mission  coordination  principles.  Third,  this 
approach  allows  us  to  immediately  fold  back  the  lessons  learned  into  the  training  program,  without  the  lag  time  often 
associated  with  laboratory  research  efforts. 

Rating  Procedures 

Each  crewmember’s  demonstrated  behaviors  across  several  coordination  subprocess  areas  are  individually  rated  by 
a  trained  observer,  using  a  1  (low)  to  5  (high)  Likert  scale.  The  process  of  providing  SME  ratings  for  each 
crewmember  across  CRM  functional  subprocesses  is  repeated  across  each  of  the  identified  mission  phases.  This 
method  of  ratings  assignment  was  developed  for  specific  use  within  the  T-MOT,  to  provide  assessments  across 
coordination  areas,  crewmember  positions,  and  mission  phases  (see  Figure  1).  These  techniques  provide  a 
circumscribed  structure  and  rules  under  which  observers  assign  numbers  to  an  attribute.  Unlike  unstructured 
obsen/ations,  this  structured  observation  technique  arms  the  observer  with  rules  to  record  specific  observations. 
The  T-MOT  employs  this  method  as  its  primary  technique  to  produce  both  quantitative  and  qualitative  data  regarding 
individual  and  team  coordination. 

In  the  T-MOT,  rating  assignments  are  supplemented  by  applying  a  unique  derivative  of  the  BARS  approach  to 
assess  team  coordination  quality.  After  extensive  testing  and  SME  review  and  analysis,  a  series  of  referent 
descriptors  for  each  mission  phase  and  coordination  subprocess  area  was  developed.  Specifically,  the  observer  is 
prompted  to  circle  either  a  “YES”  or  “NO”  response  to  a  descriptor  question.  This  method  then  elicits  a  written 
rationale  descriptor  statement  to  promote  “why”  that  item  was  circled.  In  reverse,  these  descriptive  statements 
become  the  referent  anchors  of  the  particular  rating  awarded. 
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THE  T-MOT  METHOD 


The  following  section  illustrates  the  T-MOT  method,  along  with  sample  descriptive  data  from  observed  MC-130P 
aircrews. 

Figure  1  represents  an  example  of  SME  ratings  provided  in  the  MP  phase.  Each  of  the  matrix  blocks  describe  an 
individual  or  crew  rating  received  across  each  coordination  subprocess.  The  norm  rating  for  each  of  the  matrix  cells 
is  considered  to  be  a  “3,”  and  if  any  block  is  left  blank,  the  assumed  score  is  recorded  as  such.  If  a  notable  positive 
or  negative  event  occurs,  however,  the  corresponding  block  is  scored  appropriately.  This  strategy  is  used  to  score 
those  “record  by  exception”  key  behaviors  that  occur  in  the  particular  mission  phase.  This  methodology  is  repeated 
for  each  of  the  four  remaining  mission  phases  (i.e.,  LL.  AR,  AD,  IE). 


MISSION  PREPARATION 

AC 

CP 

LN 

RN 

FE 

CSO 

CREW 

1. 

Situation  Awareness  (SA) 

4 

4 

5 

- 

- 

- 

4 

2. 

Function  Allocation  (FA) 

- 

4 

- 

4 

•  4 

2 

3. 

Tactics  Employment  (TE) 

4 

- 

5 

4 

2 

1 

4 

4. 

Time  Management  (TM) 

5 

4 

- 

4 

- 

2 

4 

5. 

Command,  Control,  &  Comm.  (C3) 

- 

- 

- 

- 

- 

4 

- 

Figure  1.  SME  Ratings  Matrix  for  the  Mission  Preparation  (MP)  Phase 


Figure  2  describes  how  two  crewmembers’  observed  social  interaction  patterns  during  MP  may  have  caused 
problems  in  another  crewmember’s  ability  to  perform  necessary  tasks  (c).  This  item  might  help  explain  a  lower 
individual  or  team  SA  score  in  one  or  more  affected  mission  phases. 


1 .0  (SA)  The  AC  will  (typically)  pay  strict  attention  to  overall  mission  preparation  details  (e.g.,  knowing  everybody’s  job, 
and  checking  other  crewmembers’  work  efforts). 

a.  Did  the  AC  pay  attention  to  detail?  . yeS  /  NO 

(Explain)  AC  constantly  checked  other's  work  efforts,  cross-referencing  to  own  mission  execution  plan _ ^ 

b.  Did  the  AC  promote  an  environment  for  open  “team”  communications?  .  YES  /  NO 

(Explain)  AC  periodically  asked  each  CM  to  (summary)  brief  others  on  current  efforts/status  of  tasks _ . 

c.  Did  non-operational  factors  (such  as  social  interaction)  interfere  with  any  crewmember’s 

time  to  perform  necessary  tasks?  . yES  /  NO 

(Explain)  FE  &  CSO  distracted  others  attention  span  on  several  occasions  with  jokes,  detractors,  etc. _ . 


Figure  2.  Mission  Preparation  (MP)  Phase  (Example) 


Figure  3  represents  a  C3  measurement  item  scored  during  the  LL  tactical  operations  mission  phase.  This  item 
describes  an  apparent  observed  strength  on  the  part  of  the  CSO  to  filter  critical  communications,  that  may  have 
contributed  to  effective  team  coordination  and  task  accomplishment. 


2.0  (C3)  CSO  receives  incoming  (distracter)  message.  The  crew  should  spend  minimum  time  dealing  with  prob!em(including 
time  for  CSO  to  filter  info.).  There  need  not  be  an  excessive  amount  of  discussion  about  the  problem’s  solution. 

a.  Was  this  event  handled  by  one  focal  crewmember,  or  was  a  full-crew  emphasis  used?  . YES  /  NO 

(Explain)  CSO  received  &  processed  the  messape  so  that  “crew”  were  aware  of  entire  situation  &  options 

b.  Did  the  CSO  filter  the  message  appropriately  ?  . yES  I  NO 

(Explain)  CSO  requested  and  received  clarifying  information  prior  to  passing  message  to  crew  for  processing  . 

c.  Were  reasonable  options  presented  for  dealing  with  the  message?  . YES  /  NO 

(Explain)  Crew  was  able  to  consider  options/solutions  because  the  CSO  clearly  filtered  &  relayed  the  SITREP . 

d.  Was  an  appropriate  decision  (outcome)  ultimately  concluded?  . YES  /  NO 

(Explain)  AC.  with  crew’s  concurrence,  decided  to  continue  mission.  Requested  CSO  relay  decision  to  C&C 


Figure  3.  Low-Level  (LL)  Tactical  Operations  (Example) 
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Figure  4  represents  a  TE  measurement  item,  demonstrated  during  the  AR  operations  mission  phase.  This  Item 
describes  the  occurrence  of  a  key  behavior  that  indicates  weakness  on  the  part  of  a  RN,  and  which  may  have 
compromised  effective  team  coordination  and  task  accomplishment. 


3.0  (TE)  The  AR  should  be  completed  early,  so  they  can  escape  hostile  airspace  quicker.  This  also  gives  the 
crew  additional  flex  time  for  later  in  the  mission,  when  mission  events  get  tight. 

a.  Did  the  crew  exercise  proper  tactical  refueling  (phase)  management  procedures?  .  YES  /  NO 

(Explain)  LN  and  RN  recognized  hostile  threat  situation,  modified  refuel  alt  to  avoid/escape  detection _ ^ 

b.  ARCP  ATA  02:35:25  .  -  Acceptable?  . YES  /  NO 

(Explain)  RN  distracted  LN  with  target  information,  causing  time  control  to  suffer.  LN  recovered  quickly _ , 

EAR  Time  02:45.55  .  -  Acceptable?  . YES  /  NO 

(Explain)  Time  versus  fuel  transfer  I  AW  Dash-1  specs.  No  delay  caused  by  crew  coordination  issues _ , 

Figure  4.  Aerial  Refueling  (AR)  Operations  (Example) 


c. 


Figure  5  represents  an  FA  measurement  item  during  the  AD  operations  mission  phase.  This  item  describes  a 
possible  strength  on  the  part  of  the  FE  to  prompt  for  checklist  responses.  It  is  reasonable  to  assume  that  this 
behavior  may  have  contributed  to  effective  team  coordination  and  task  accomplishment. 


4.0  (FA)  The  LN  and  AC  (combined)  typically  make  the  decision  to  command  the  AirDrop.  A  “blind”  drop  was  specified  in  the 
mission  planning  phase  -  the  crew  will  need  to  remember  this  point.  Time  compression,  poor  visibility,  and  mission 
environment  may  cloud  their  decision  to  drop  (the  DZ  will  likely  not  be  visible  until  the  final  few  second  of  the  run-in 
procedure. 

a.  Was  the  final  decision  to  AirDrop  coordinated  properly?  . YES  /  NO 

(Explain)  All  crewmembers  verbally  indicated  to  AC,  with  CP  backup,  that  DZ  was  positively  iPU  and  in  sight  . 

b.  Were  there  any  problems  noted  during  the  AirDrop  (process)?  . YES  /  NO 

(Explain)  Slow  responses  to  multiple  checklist  items.  Several  FE  prompts  for  responses.  Operation  completed. 

Figure  5.  AirDrop  (AD)  Operations  (Example) 

Figure  6  represents  an  SA  measurement  item  recorded  during  an  IE  operations  mission  phase.  This  item  illustrates 
that  a  SA  weakness  exists  on  the  part  of  the  CP  in  incorrectly  identifying  the  Landing  Zone  (LZ).  This  behavior  may 
have  compromised  effective  team  coordination  and  task  accomplishment. 


5.0  (SA)  The  RN  (FLIR,  SCNS),  AC  (NVG  Visual),  LN  (Radar,  SCNS)  will  need  to  demonstrate  a  high  level  of  effective 

coordination  when  working  together  to  ID  the  airfield  for  the  covert  Infil  operation.  CP  also  needs  to  assist  with  high- 
precision  chart  interpretation,  effective  visual  NVG  scanning  techniques,  and  efficient  communications  to  other 
crewmembers. 

a.  Did  the  crew  work  together  to  ID  the  LZ  airfield? . YES  /  NO 

(Explain)  The  LN  and  RN  confirmed  to  each  other,  then  crew,  that  Nav  systems  were  accurate,  and  L7  in  sipht  . 

b.  Did  the  crew  have  problems  with  the  approach  (process)?  .  YES  /  NO 

(Explain)  CP  informed  crew  that  LZ  in  sight  (bad  call),  major  (late)  course  corrections  rea'd.  to  compensate 

Figure  6.  Infil/Exfil  (IE)  Operations  (Example) 

Figure  7  represents  an  SA  measurement  item.  This  item  illustrates  the  methodology  for  recording  observed  SA 
behaviors  during,  for  example,  the  MP  phase.  This  example  portrays  one  crewmember’s  (RN)  integration  of  “real 
world”  aspects  of  the  missions  Area  of  Operations  (AO)  into  the  CMT  scenario,  possibly  explaining  the  development 
or  promotion  of  an  overall  enhanced  team  SA. 


1 .0  (SA)  At  least  one  crewmember’s  overall  SA  should  be  high,  and  an  assessment  of  mission  difficulty  should  be 
made  based  on  (for  example):  marginal  WX,  threat  saturation  is  high,  large  no.  of  mission  events,  etc. 

a.  Did  any  individual  crewmember  indicate  an  overall  assessment  of  mission  difficulty?  . YES  /  NO 

(Explain)  RN  indicated  '\..this  mission  is  just  like  our  everyday  operations../’  (He  has  operated  in  same  AO) 

b.  Did  crewmember(s)  prepare  for  unexpected  or  contingency  situations?  . YES  /  NO 

(Explain)  RN  added  several  “real  world"  aspects  of  INTEL  &  contingency  plans  to  mission  execution  plan 

Figure  7.  Situation  Awareness  (SA)  Functional  CRM  Subprocess  Area  (Example) 
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Figure  8  represents  an  FA  measurement  Item,  recorded  during  the  MP  phase.  In  this  example,  the  AC  appears  to 
be  the  driving  force  in  developing  a  task  orientation  to  balancing  and  completing  all  mission  preparation  events.  This 
demonstration  of  key  leadership  behavior  may  be  illustrative  of  a  higher  level  of  FA,  and  effective  team  coordination. 


2.0  (FA)  Workload  and/or  task  distribution  should  be  clearly  communicated  and  acknowledged  by  crewmembers. 

a.  Was  the  mission  workload  distribution  clearly  communicated  and  acknowledged?  .  YES  /  NO 

(Explain)  AC  assigned  primary  task  distribution  for  each  CM,  presented  options  to  complete  secondary  tasks 

b.  Were  secondary  tasks  prioritized  so  as  to  allow  sufficient  resources  for  primary  tasks? . YES  /  NO 

(Explain)  Crew  was  unable  to  grasp  secondary  tasks  due  to  intense  workload  -  focus  was  on  primary  tasks  only. 

c.  Did  non-operational  factors  (such  as  social  Interaction)  interfere  with  any  crewmember’s 

abilities  while  performing  necessary  tasks?  . YES  /  NO 

(Explain)  This  crew  was  totally  task  focused  >  NO  social  interaction  observed  (AC  set  example  for  crew)  . 


Figure  8.  Function  Allocation  (FA)  Functional  CRM  Subprocess  Area  (Example) 


Figure  9  represents  a  TE  measurement  item  describing  observed  behaviors  in  the  MP  phase.  In  this  example,  the 
LN,  RN,  and  CP  developed  a  pragmatic  tactics  employment  plan.  This  crew’s  use  of  assumptions  and  deceptiori,  for 
example,  in  their  mission  execution  tactics  is  illustrative  of  an  extremely  higher  level  of  TE  and  effective  overall  team 
coordination. 


3.0  (TE)  There  are  (typically)  three  tactical  options  to  use  in  order  to  go  undetected:  Altitude,  Airspeed,  and  Terrain. 

a.  Was  a  particular  mix  of  tactics  options  considered?  . yES  /  NO 

(Explain)  LN  and  CP  developed  an  assumption  that  threat  detection  was  100%  -  opted  for  deception  tactics 

b.  Did  the  crew  change  the  tactics  options  as  a  function  of  difficulty  in  each  mission  phase?  . YES  /  NO 

(Explain)  LN  RN  developed  an  altitude  masking,  tactical  LL  altitudes.  &  deception  tactics  plans.  asTiecessary . 

c.  Was  one  option  (e.g.,  speed)  preferred  over  the  others?  . yeS  /  NO 

(Explain)  Primary  tactic  was  LL  contour  altitude  masking  using  terrain  and  other  cultural  features  _ , 

d.  Did  any  crewmember  periodically  review  or  verify  the  status  of  the  threat  planning  strategy?  . YES  f  NO 

(Explain)  CP  cross-checked,  Questioned,  and  referenced  LN,  RN  durinp  execution  plan  development _ ^ 


Figure  9.  Tactics  Employment  (TE)  Functional  CRM  Subprocess  Area  (Example) 


Figure  10  illustrates  the  measurement  of  exceptional  TM  behaviors  observed  during  MP.  In  this  example,  the  AC 
negotiated  a  plan  for  the  effective  use  of  time,  as  a  resource.  Other  crewmembers  responded  to  the  AC’s  plan  in  a 
positive  manner,  but  the  resultant  effect  appeared  to  be  only  partly  successful.  This  may  be  illustrative  of  a  higher 
level  of  TM,  but  not  one  that  was  a  contributor  to  effective  crew  coordination. 


4.0 


a. 


(TM)  An  end-mission  planning  time  should  be  indicated  up  front  -  most  likely  by  an  emergent  "leader.” 

Did  any  crewmember  indicate  the  need  for  an  end-mission  planning  time?  . YES  /  NO 

(Explain)  AC  negotiated  time  read  to  complete  planning  tasks,  and  established  the  mission  briefings  start  time. 

Was  that  time  noted  by  all  other  crewmembers?  . 

(Explain)  LN,  RN  indicated  that  “..Af  we’re  not  done,  we  will  continue  workinp  durinp  the  briefs../' 

Did  any  crewmember  designate  activities  to  establish  a  proper  balance  between 

their  own  authority,  time  available,  and  crewmember  participation?  . YES 

(Explain)  E^ch  CM  focused  on  own  tasks,  no  observed  willingness  to  help  other's  complete  their  tasks _ . 

Was  adequate  mission  preparation  time  allocated  for  a  comprehensive  pre-mission  briefing?  .  YES  /  NO 

(Explain)  AC  allowed  brief  start  time  to  pass  as  not  everyone  was  ready.  Result  -  incomplete  &  hurried  briefims. 


YES  I  NO 


/  NO 


d. 


Figure  10.  Time  Management  (TM)  Functional  CRM  Subprocess  Area  (Example) 
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Figure  11  illustrates  the  measurement  of  C3  behaviors  observed  during  the  MP  phase.  In  this  example,  the  crew 
appeared  to  be  unwilling  to  operate  “out  of  their  own  box.”  They  would  not  request  specific  external  resources,  nor 
were  they  willing  to  make  mission  assumptions.  This  may  be  illustrative  of  a  lesser  developed  level  of  C3,  and  may 
possibly  explain  a  lack  of  effective  crew  coordination. 


5.0  (C3)  Crew's  willingness  to  challenge  the  system. 

a.  Do  crewmembers  request  specific  (Internal/External)  resources  they  need?  . YES  /  NO 

(Explain)  Crew  was  unwHIing  to  "operate  outside  of  their  own  resources.  ”  No  repuests  for  other  ‘'players*’ 

b.  Do  crewmembers  question/challenge  assumptions  (e.g.,  within  frag,  threat  SITREP,  etc.)?  . YES  /  NO 

(Explain)  RN  questions  difficulty  of  AD  operation,  as  tasked  -  Alternative  suppestions  offered  to  assure  success. 

c.  Do  crewmembers  ferret  out  needed  materials  and  information  from  alt  sources?  . YES  /  NO 

(Explain)  Unless  info  was  specified  in  FRAG,  crew  was  unwillinQ  to  make  assumptions,  or  request  clarification  . 


Figure  11.  Command,  Control,  and  Communications  (C3)  Functional  CRM  Subprocess  Area  (Example) 


T-MOT  APPLICATION 

In  our  larger  research  program,  the  T-MOT  supports  two  major  areas  of  Inquiry  concerning  team  mission 
performance,  coordination  processes,  and  their  relationships  to  each  other; 

Research  Questions 

One,  does  team  coordination  affect  mission  performance?  Our  primary  research  interest  is  to  demonstrate  a 
positive  relationship  between  overall  team  coordination  and  tactical  mission  performance.  While  the  unequivocal 
documentation  of  this  linkage  will  provide  a  valuable  addition  to  the  team  effectiveness  literature,  our  conceptual 
approach  permits  us  to  scrutinize  this  relationship  in  further  detail  (Spiker,  Tourville,  Silverman,  &  Nullmeyer,  1996). 
Once  coordination-performance  relationships  have  been  identified,  we  intend  to  probe  the  data  further,  to  determine 
whether  effective  aircrews  exhibit  a  consistent  set  of  behaviors  that  can  be  "captured"  and  provided  as  feedback 
during  training. 

One  way  to  accomplish  this  is  to  compute  “derived  indices”  from  the  key  behaviors  in  the  T-MOT  checklist.  For 
example,  we  might  find  that  the  most  effective  aircrews  tend  to  exhibit  some  consistency  in  terms  of:  (a)  having  the 
AC  serve  as  the  leader  (as  opposed  to  some  other  crew  position  or  none),  (b)  making  effective  use  of  the  CP  during 
mission  preparation,  or  (c)  early  recognition  by  one  or  both  navigators  of  problems  In  the  designated  control  times, 
making  the  mission  difficult. 

Once  these  “derived  indices”  have  been  identified,  a  descriptive  profile  of  a  crew’s  relative  standing  on  each  index 
may  be  determined.  For  example,  we  might  rank  a  crew  either  a  1,  2,  or  3  on  the  AC  index,  depending  on  whether: 
the  AC  served  as  an  effective  leader  (3),  some  other  crewmember  (e.g.,  LN  or  RN)  was  the  leader  (2),  or  there  was 
no  identified  leader  (1).  Similar  constructions  are  then  developed  for  the  other  indices,  yielding  a  crew  descriptive 
profile,  or  vector,  such  as  [3,  2,  3.  1,  etc.],  wherein  the  dimensionality  of  the  vector  would  be  determined  by  the 
number  of  “derived  indices”  that  were  identified. 

Two,  how  do  different  crew  positions  affect  team  coordination  and  tactical  mission  performance?  The  T-MOT 
data  matrix  encompassing  the  five  CRM  functional  subprocess  areas  and  five  SOF  tactical  mission  phases  may  be 
expanded  to  isolate  behaviors  associated  within  the  six  Individual  crew  positions.  This  would  allow  an  assessment 
of  the  contributions  of  each  crew  position  to  overall  team  coordination  and  its  component  subprocesses,  as  well  as 
overall  team  mission  performance  and  its  scenario  elements.  The  expansion  of  this  matrix  would  allow  four 
additional  questions  to  be  addressed: 

First,  which  crew  positions  have  the  strongest  relationship  to  overall  team  coordination?  While  it  is  possible  that  all 
positions  are  equally  vital  in  supporting  the  "emergence"  of  an  effective  team,  the  realities  of  operating  the  MC-130P 
may  be  such  that  some  positions  play  a  larger  role  than  others.  For  example,  we  may  expect  to  see  that  the  AC, 
CSC,  and  two  navigator  positions  (LN,  RN),  by  virtue  of  their  multiple  tasking  and  extensive  communications 
requirements,  have  a  larger  impact  on  team  coordination  ratings  than  do  the  FE  or  CP. 

Second,  an  even  more  involved  set  of  questions  concerns  the  differential  impact  of  crew  position  on  specific  team 
coordination  areas.  For  example,  a  good  LN  may  rate  high  on  SA  and  TM,  but  TE,  FA,  and  C3  may  not  be  as 
important.  On  the  other  hand,  a  good  CSO  may  rate  high  on  TM  and  C3,  but  lower  on  TE,  FA,  and  SA.  While  such 
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determinations  can  become  somewhat  involved  (i.e.,  six  crew  positions  by  five  coordination  areas),  they  hold 
considerable  potential  for  helping  to  identify  the  content  of  future  training  interventions  (e.g.,  specialized  training 
workstations,  communications  checklists). 

Third,  do  some  crew  positions  play  a  larger  role  than  others  in  overall  team  mission  performance?  Povenmire  et  al. 
(1989)  obsen/ed  that  the  squadron-provided  rankings  of  B-52  crew  mission  performance  were  primarily  influenced 
by  the  skill  level  of  the  Radar  Navigator.  Based  on  anecdotal  accounts,  a  similar  pattern  may  exist  in  MC-130P 
crews,  with  the  LN's  behavior  being  particularly  central  to  mission  performance. 

Fourth,  does  the  influence  of  specific  crew  positions  vary  when  overall  mission  performance  is  divided  into  its 
specific  scenario  elements?  For  example,  an  AR  operation  is  highly  dependent  on  the  ability  of  the  FE  to  calculate 
fuel  transfer  and  monitor  systems  (e.g.,  hoses)  during  transfer.  At  the  same  time,  the  CSO  must  send,  receive,  and 
transfer  (secure)  messages  to  C&C  authorities,  in  order  to  coordinate  the  AR  operation  with  the  receiving  party. 
Moreover,  one  must  consider  the  additional  labor  of  the  pilots  and  the  navigators  required  to  ensure  AR  success. 
Despite  this  obvious  team  effort,  it  may  be  that  the  critical  role  of  the  CSO  in  coordinating  this  particular  task  is  the 
most  heavily  weighted  determinant  of  overall  team  AR  performance. 

Methodological  Constraints 

The  T-MOT  approach  helps  overcome  five  methodological  constraints  that  plague  traditional  team  studies;  (a)  small 
sample  sizes,  (b)  simulator  reliability,  (c)  scenario  realism,  (d)  limited  access  to  simulators,  and  (e)  uncontrolled 
scenario  variability. 

Small  Sample  Size.  Several  factors  can  necessitate  a  small  sample  size  when  conducting  team  coordination 
research  in  realistic,  operational  settings.  In  our  case,  the  small  relative  size  of  the  SOF  operational  population  is  a 
major  driver  behind  a  small  sample  size.  Researchers  must  accordingly  develop  their  programs  in  conjunction  with 
ongoing  training  and  acquire  subjects  where  available. 

The  analytic  approach  of  the  T-MOT  to  mitigate  the  risk  of  small  samples  is  to  look  primarily  for  powerful  effects 
rather  than  subtle  ones.  For  example,  in  combination  with  crew-based  measures  of  coordination  process,  the  T- 
MOT  supports  collecting  ratings  and  observations  across  ail  crew  positions  and  several  key  dyads  (FE  and  CSO) 
and  triads  (LN,  RN,  and  AC).  This  enables  additional  comparisons  and,  in  some  cases,  significantly  increases  the 
sample  size. 

Simulator  Reliability.  Whether  conducting  training  or  research  in  simulators,  one  obstacle  that  must  eventually 
be  confronted  is  simulator  reliability.  Advanced  technology  is  not  perfect,  particularly  as  it  involves  complex 
computational  systems. 

The  T-MOT  uses  two  strategies  to  minimize  the  deleterious  effects  of  simulator  reliability.  First,  we  monitor  multiple 
phases  of  flight  that  involve  elements  having  redundant  tactical  and  technical  significance.  Within  the  simulator 
scenario,  for  example,  there  are  multiple  Infil/Exfils.  If  the  simulator  malfunctions  during  the  course  of  training,  we 
should  collect  information  on  at  least  one  of  these  events.  Second,  since  our  data  collection  strategy  covers  the 
entire  mission,  from  mission  preparation  through  mission  execution  and  debrief,  we  are  guaranteed  process  data  on 
every  crew  observed,  even  in  the  extreme  case  of  the  simulator  not  running  at  all. 

Scenario  Realism.  This  constraint  poses  a  significant  challenge  to  team  performance  measurement.  Despite 
the  high  fidelity  of  the  WST,  aircrews  often  fail  to  treat  simulator  training  with  the  same  seriousness  as  they  treat 
aircraft  operations.  This  effect  may  even  be  compounded  by  instructors  with  similar  views. 

The  T-MOT  was  designed  with  these  factors  in  mind.  We  have  limited  the  amount  of  direct  aircrew  input,  and  our 
observations  are  designed  to  be  as  unobtrusive  as  possible  so  as  to  integrate  smoothly  with  the  normal  flow  of 
training  events.  To  reduce  the  potential  effects  of  instructor  skepticism  and  variation,  a  researcher  with  high 
credibility  among  the  aircrew  due  to  previous  experience  serves  as  a  participant-observer.  He  guides  the 
introduction  of  the  scenario  to  ensure  its  professional  and  realistic  presentation,  and  often  role-plays  (along  with 
instructors)  responses  and  directives  from  other  agencies  that  would  be  included  if  this  were  a  “real-world”  mission. 

Limited  Access  to  Simulator.  Due  to  limited  available  space  in  most  WSTs,  it  is  difficult  for  researchers  to  have 
direct  access  when  making  team  coordination  observations.  Simulators  are  designed  to  match  the  aircraft  cockpit 
which  do  not  usually  have  additional  seats  for  observers.  The  “extra”  spaces  that  are  provided  in  simulators  usually 
accommodate  instructors.  Additionally,  making  over-the-shouider  obsen/ations  while  standing  is  not  permitted  in 
simulators  with  functioning  motion  systems  due  to  safety  concerns. 
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The  T-MOT  addresses  this  problem  by  focusing  on  those  parts  of  the  process  that  are  reliably  available,  such  as 
mission  planning,  briefings,  and  communication,  during  both  mission  preparation  and  execution.  These  are 
invaluable  sources  of  data  for  which  data  may  be  consistently  collected. 

Uncontrolled  Scenario  Variability.  Uncontrolled  scenario  variability  is  a  natural  characteristic  of  fluid  training 
environments,  where  training  events,  training  requirements,  aircraft  configurations,  simulator  capabilities,  and 
instructors  are  constantly  changing.  To  circumvent  this  issue  for  our  research,  an  agreement  was  established  for  a 
“limited  freeze"  in  the  current  scenario  while  data  were  collected.  This  agreement  helps  to  ensure  scenario  stability 
across  the  observed  crews,  so  that  valid  comparisons  might  be  drawn.  The  previously  mentioned  participant- 
observer,  and  his  acceptance  by  the  instructors,  also  helped  to  remove  some  of  the  variability  associated  with 
scenario  administration  that  can  occur  from  instructor  to  instructor. 

Despite  these  precautions,  our  simulator  CMT  scenario  has,  in  fact,  undergone  several  major  modifications  to  the 
mission  scope,  concept,  and  definition.  However,  the  T-MOT  withstood  each  of  these  changes,  requiring  only  minor 
modification  to  changing  data  formats.  This  is  because  the  T-MOT  approach  is  focused  more  on  the  specific 
scenario  being  flown,  and  less  on  particular  events  as  they  occur.  The  CMT  scenario  is  not  as  highly  scripted  or 
controlled  as  in  some  approaches,  but  it  is  more  appropriate  in  representing  a  fluid  training  environment  where 
instructor  turnover  is  high,  training  events  change  often,  and  mission  events  are  unlikely  to  "freeze." 

IMPLICATIONS 

A  desired  goal  of  CMT  is  to  produce  lasting,  positive  behavioral  change  that  results  in  improved  team  coordination 
effectiveness.  Data  generated  from  the  T-MOT  should  further  our  understanding  of  those  constituent  subprocesses 
required  for  effective  team  coordination.  The  results  of  this  investigation  will  produce  data  for  CMT  instructors  and 
training  developers  to  populate  aircrew  team  coordination  protocols  and  procedures  with  more  focused,  skill-oriented 
content.  Interventions  may  then  be  developed  to  improve  aircrew  coordination  subprocesses  in  task-specific 
situations  where  a  high  level  of  interdependency  is  required  among  members.  We  believe  this  approach  will  permit 
the  T-MOT  to  be  expanded  to  future  distributed  interactive  simulation  (DIS)  environments  where  other  agents 
provide  stimulus  events.  Two  impact  areas  are  foreseen: 

Application  of  the  T-MOT  to  Other  Domains.  A  recent  study  by  Brannick,  Prince,  Prince,  and  Salas  (1995) 
illustrated  the  Navy’s  similar  view  of  relationships  in  team  coordination  processes.  Consistent  with  the  present 
approach,  team  coordination  was  considered  to  be  a  logical  set  of  processes,  and  consistency  among  process  raters 
in  assigning  coordination  scores  across  mission  scenarios  was  ably  demonstrated.  This  consistency  was  taken  as 
evidence  for  the  quality  of  their  rating  methodology  and  the  selection  of  concrete  coordination  behaviors  for  direct 
observation. 

We  similarly  believe  that  the  robustness  and  sensitivity  embedded  in  the  T-MOT’s  methodological  strategy  supports 
multiservice  training  or  rehearsal  applications.  The  T-MOT’s  flexible,  real-time  use  of  objective  and  subjective 
assessment  indices  may  also  be  readily  adapted  to  Distributed  Mission  Training  (DMT)  environments,  such  as  those 
found  in  networked  simulator  mission  training  for  joint  SOF  operations,  AC-1  SOU  Gunship  operations  or  multiship 
fighter  operations. 

Development  of  a  Team  Mission  Readiness  Assessment  Tool  (T-MRAT).  The  results  of  a  study  of  CMT  team 
coordination  effectiveness  within  the  confines  of  a  WST  should  form  an  empirical  foundation  for  development  of  a 
tool  used  to  gauge  an  operational  aircrew  team’s  overall  mission  readiness  prior  to  actual  mission  execution.  The  T- 
MRAT  might  be  used  for  other  purposes  as  well,  including  aircrew  evaluation,  aewmember  evaluation,  and 
instructional  evaluation.  The  portable  data  collection  methods  used  in  this  study  will  be  specifically  designed  for 
expansion  and  incorporation  into  team  effectiveness  assessments  during  joint  mission  rehearsal  exercises  and  other 
operational  unit  activities. 
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